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Remarks 

Objection to the abstract 

The abstract was objected to because the indentation on line 5 is similar to that 
designating the start of a new paragraph. The abstract has been amended above to remove the 
extra indentation, thus obviating this ground for rejection. Applicants, thus, respectfully request 
reconsideration and withdrawal of this ground for objection. 

Claim rejections under 35 USC lOSfa^ 

(a) Claims 1-2, 4-7 and 9-10 were rejected under 35 USC § 103(a) as being unpatentable over 
Gibson-694 and Gibson-634. Notably, claims 2, 4-7 and 9-10 depend either directly or indirectly 
from claim 1, and appropriately (as discussed below) claim 3 is not amongst the claims rejected 
utilizing the Gibson references. Claim 1 has been amended to include all the limitations of claim 
3, and thus Applicants submit claim 1, as amended, is patentable over this combination of 
references. In this light, Applicants respectfully request reconsideration and withdrawal of this 
ground for rejection of amended claim 1, and claims 2, 4-7 and 9-10 that depend directly or 
indirectly from amended claim 1. 

(b) Claims 3 and 8 were rejected under 35 USC § 103(a) as being unpatentable over Gibson- 
694 in view of Gibson-634 and fiirther view of Avila, et al (USP 5,802,353). 

The presently claimed invention relates to haptic rendering that does, indeed, begin with 
the occupancy map (OPA) and chain mail (CMA) algorithms of Gibson's work. The presently 
claimed invention, however, overcomes the limitations of these algorithms as they are applied to 
PC-based, real-time haptic interactions and rendering of volumetric soft-bodied objects, and to 
developing haptic virtual reality applications without the need for any pre-computation. 

The required use of pre-computation as taught by the cited references puts a severe 
restriction on real-time applications, in which dynamic changes occur in a virtual environment, as 
is the case of deformable haptics. The method of Avila referenced in the instant Action requires 
such pre-calculation of forces. The Avila method was used by Sarah Gibson, et al, in their 1998 


Page 7 of 9 


Appl. Ser. No.: 09/677,073 
Atty. Docket No.: 12001-102 

paper (provided in A ppendix A h ereto) "Volumetric object modeling for surgical simulation" 
Medical Image Analysis, vol 2(2), pp. 121-132, 1998. In this paper, which Applicants note was 
published at least a year after the filing of each of the cited references, Gibson recognizes that 
pre-computation may be too computationally expensive for real-time interactions needed for 
haptic rendering. In the same paper, Gibson notes that they have not achieved the integration of 
haptics and deformation, stating in Section 4.2.3, . .Eventually, the force feedback device will 
provide valuable sensory feedback to the surgeon during simulation of tissue deformation and 
cutting." 

Even in Gibson's 1999 paper, "Using Linked Volumes to Model Object Collisions, 
deformation, Cutting, Carving, and Joining" {IEEE Transactions on Visualization and Computer 
Graphics 5(4), 333-348)^ it is evident that they have not been able to do perform such 
integration. 

This is all in contrast to the claimed inventive method, which uses CMA to compute, in 
real-time, the force feedback used by the haptic device, requiring no pre-computation. 

It is also important to note that papers published as late as 2002 point out the difficulty 
of extending her work to haptics (see Park, J. et al., "Alternative Modeling Scheme for Interactive 
Volume Deformation", Proceedings of Korea-Germany Joint Workshop on Advanced Medical 
Image Processing, Heidelberg, Germany, 2002.)^ 

In light of the foregoing. Applicants respectfully submit that claims 1-10 stand in a 
condition for allowance, and a notice to that effect is eamestly solicited. If questions remain, 
please call Applicants' attorney at the number given above. 

Respectfully submitted, 

BHARTI TEMKIN, et al., 
Applicants 

^ this journal articles will be sent under separate cover to the USPTO as soon as obtained by 
Applicants* attomey 

^ this journal articles will be sent under separate cover to the USPTO as soon as obtained by 
Applicants' attomey 
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Abstract 

Surgical simulation has many applications in medical education, surgical training, surgical plan- 
ning and intra-operative assistance. However, extending current surface-based computer graphics 
methods to model phenomena such as the deformation, cutting, tearing or repairing of soft tissues 
poses significant challenges for real-time interactions. This paper discusses the use of volumetric 
n nethods for modeling complex anatomy and tissue intftrantinny^, New techniques are introduced 
at use volumetric methods for modeling soft-tissue deformation and tissue cuttimg at interactive 
rates. An initial prototype for simulating arthroscopic knee surgery is described which uses 
volumetric models of the knee derived from 3-D magnetic resonance imaging, visual feedback 
via real-time volume and polygon rendering, and haptic feedback provided by a force-feedback 
device. 

Keywords: surgical simulation, volume graphics, volumetric modeling 
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1. INTRODUCTION 

Computer-based surgical simulation has many applications in 
medical education, surgical training, surgical plarming and 
intra-operative assistance. In education and training, surgical 
simulation can reduce costs associated with cadaver speci- 
mens, provide experience with a greater variety of pathologies 
and complications, and provide the ability to repeat or replay 
training procedures. In surgical planning, simulators can 
enable rehearsal of difficult procedures or planning on patient- 
specific anatomy and can enhance communication among 
medical professionals or between doctors and their patients. 
Intra-operatively, computer modeling can aid in navigation 
by augmenting the Umited surgical field with a more global 
view of a patient's anatomy. For example, computer modeling 
-^ould provide guidance by preventing the surgical instrument 
om entering sensitive regions. 

•Corresponding author 
(e-mail: gibson@merl.com) 


In order to provide useful feedback to the user, surgical 
simulators must provide adequate realism. Tissue models 
should respond in a realistic way when they are manipulated. 
Rendered images of the surgical field must be realistic enough 
to be compelling. Haptic or force feedback must mimic forces 
experienced in real-life because the sense of touch provides 
important cues in surgery. These requirements impose sig- 
nificant demands on the surgical simulator. These challenges 
include a need for physicaUy realistic modeling (such as soft- 
tissue deformation and tissue cutting or tearing) and tradeoffs 
among physical and visual realism, the need for real-time 
interaction and cost 

In computer graphics, objects are conunonly represented 
by surface-based polygonal models. Because graphics 
workstations have special-purpose hardware for fast polygon 
rendering and because algorithms and systems have been 
developed for modeling physical interactions between (rigid) 
polygonal objects, currently there are advantages to using 
polygonal models in a surgical simulator. However, because 
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Closed Feedback Loop 


Figure 1. Surgical simulation system components. This system 
provides a closed feedback loop between the user and the simulator. 
The user can manipulate object models and observe the results both 
visually and haptically. 

they cannot represent inteirior structure, surface-based 
models are inadequate for modeling objects and tissues 
with complex interiors, for modeling the deformation of 
arbitrary or heterogeneous volumes and for simulating 
cutting through tissues. In addition, surface-based methods 
cannot model fluid flow, tissue tearing or bone fracturing. 
In contrast, volumetric models can incorporate a great deal 
of information about interior structure and the mechanical 
properties of heterogeneous tissues. By adjusting forces 
that hold volumetric elements together, it may be possible 
to model fluids, tearing and fracturing using volumeUic 
models. In this paper, we discuss techniques for deforming 
volumetric object models, for detecting collisions between 
volumetric objects and for modeling tissue cutting. In 
addition, we describe an initial prototype system for 
simulating arthroscopic knee surgery. 

The long-term goal of this project is a computer-based 
surgical simulation system that uses models generated from 
patient specific data. As illustrated in Figure 1, the system 
will provide both visual and haptic feedback to the user via 
real-time rendering and a force-feedback device. Physically 
realistic interactions between anatomical models and user- 
controlled surgical instruments will provide a closed feedback 
loop between the user and the simulator. Important tech- 
nologies required by the simulator include: image acquisition 
and segmentation of patient-specific 3-D data to generate 
object models; visual and haptic feedback through real-time 
rendering and an electromechanical force-feedback device; 
manipulation of visual parameters and physically realistic 
interactions with the model through a haptic interface. 

2. PRIOR WORK 

There are several surgical simulation systems under 
development both conmiercially and in research laboratories. 


For example, a number of endoscopy simulators enable 
navigation through stationary object models (Vining et ai, 
1993, 1994; Geiger and Kikinis. 1995; Hong et al., 1995; 
Lorensen et a/., 1995; Ziegler et at., 1995). In some 
navigation or virtual fly-throughs, path planning and collision 
avoidance between the virtual endoscope tip and static 
object surfaces are incorporated into the models (Geiger 
and Kikinis, 1995; Hong et al., 1995; Lorensen et al., 

1995) . Deformation of surface models is used to model 
soft-tissues by some surgical simulations systems (e.g. Cover 
et aL, 1993; Kuhnapfel et al., 1997). These systems use 
either mass-spring models or control points on spline-based 
surfaces for interactive deformation of surface-based object 
models. 

A number of groups have used volumetric methods for 
modeling deformation and cutting of tissue volumes. Finite- 
element analysis has been applied to facial and muscle mod- 
eHng (Waters, 1987; Terzopoulos and Waters, 1990; Chen, 
1991), and in surgical simulation (e.g. Pieper, 1992; Hunter 
etaL, 1993; Cotin etal, 1996). Manipulation of voxel-based 
objects (Kaufman, 1996) has been used in object modeling 
(Gibson, 1995) and combined with force feedback for haptic 
exploration of voxel-based objects (Avila and Sobierajski. 

1996) . Sculpting of volumetric objects is described in Wang 
and Kaufman (1995) and Galyean and Hughes (1991). 

3. VOLUMETRIC METHODS FOR SURGICAL 
SIMULATION 

Modem medical image scanners can provide a great deal of 
information about internal anatomical structure and function 
in individual patients. Figure 2 shows a single slice of a 3-D 
magnetic resonance imaging (MRI) image with resolution 
0.25 X 0.25 X 1.4 mra^. Internal structure, such as the 
heterogeneous muscle and fatty interstitial tissues indicated 
by am)ws in the image, can be important for modeling tissue 
biomechanics. In surface-based methods, this internal detail 
is discarded and an object is modeled by a hollow sheU. Vol- 
umetric object representations provide a way of incorporating 
all of the information available from the MRI data into a 
functioning anatomical model. 

3.1. Volumetric representation 

The volumetric object is represented by a 3-D array of sam- 
pled data points. The data can be either regularly or irregularly 
spaced. Examples of regularly spaced data include image 
volumes generated by 3-D medical scanners such as MRI or 
computed tomography (CT) devices. Examples of irregular 
volumes include data at the nodes of a finite-element simula- 
tion or data samples from geological surveys. 
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Figure 2. Single plane from a high-resolution MRI image volume 
with voxel resolution: 0.25 x 0.25 x 1.4 mm^. Arrows indicate 
tructure in muscle and fatty tissue that are important in modeling 
tissue behavior but that are hard to incorporate into surface-based 
models. 

Elements in the volume encode information about visual 
and/or physical properties of the object. Each element can 
be represented by a single value, such as the sampled image 
intensity, or a more complex data structure that encodes infor- 
mation such as color, transparency, edge strength, plasticity, 
tissue type and connections to neighboring elements. The 
ability to encode many different attributes for each element 
enables the modeling of complex materials, structures and 
behaviors. However, since volumetric objects can consist 
of thousands to millions of elements (an MRI image of size 
256x256x256 contains 16 million voxels) this representation 
poses significant challenges for rendering, data storage and 
retrieval and tissue modeling. Some approaches for dealing 
with these large volumes in rendering and tissue modeling 
are presented here. Future research into hierarchical data 
representations and data compression will make volumetric 
methods even more practical for real-time apphcations. 

3.2* Soft-tissue deformatioD with volumetric models 

5.2.7. Mass-spring systems and linear finite-element 
methods 

The two most common methods used for modeling volumet- 
ric deformation are mass-spring systems and linear finite- 
element methods (FEM). Though mass-spring systems and 


hnear FEM methods are different techniques, in a static sim- 
ulation both result in a large system of linear equations of the 
form 

Ku=F, 

where K is the object stiffness matrix (a function of material 
elastic properties which depend on the method), u is the 
vector representing element displacements and F is a vector 
representing internal and external forces in the system. In a 
dynamic simulation, where changes in the shape are observed 
as the object moves towards equilibrium, the effect of inertial 
and damping forces are modeled using a system of second- 
order differential equations: 

Mii-\-Cu +Ku=F 

where M and C are mass and damping matrices respectively, 
and ii and ii are the first- and second-order derivatives of u 
with respect to time. In a dynamic computer simulation, this 
system is evolved through time using numerical techniques. 

In both static and dynamic simulations, solving for the 
displacements, is computationally demanding for a large 
system. The interaction speed can be increased by a sig- 
nificant reduction in the number of elements in the system 
(Hunter et aL, 1993) and/or pre-processing of the data. Bro- 
Nielsen simpUfies the system by solving for displacements 
of only surface elements and by assuming a small number 
of externally applied forces (Bro-Nielsen, 1997). Pendand 
and Williams (1989) pre-calculate the deformation modes of a 
given object and calculate deformations for an arbitrary force 
as a superposition of these deformation modes. Bro-Nielsen 
and Cotin (Bro-Nielsen, 1995; Bro-Nielsen and Cotin, 1996; 
Cotin et ai, 1996) use a similar method, pre-calculating re- 
sponses to infinitesimal forces and deformations for each node 
in the element and then approximating the global deformation 
as a hnear superposition of these pre-calculated responses. 

These methods have a number of limitations. First, even 
with significant preprocessing, the size of the deformation is 
limited and the number of elements must remzun relatively 
small for real-time modeling. Cotin et aL use the above 
method in a system that can deform a liver model with 8000 
elements at 15 frames per second*. This is still a relatively 
small number of elements, especially if the complex structure 
of internal vessels and ducts are modeled. In addition, the 
system requires hours of preprocessing which prohibits inter- 
active changes in topology due to tissue cutting, tearing or 
sumring. 

Second, the assumptions of linearity and superposition are 
not valid for human tissue. Figure 3 shows a typical stretch 
versus appUed force curve for human tissue [see Fung (1993) 

■personal conununication from Stephan Colin. April, 1997. 


124 


S. Gibson et al 



Strain / Stretch / Deformation 


Figure 3. Typical stress-strain or force-stretch curve for human 
tissues [see Fung (1993) for examples]. Note that the relationship is 
nonlinear and that the system undergoes hysteresis (the force-stretch 
curve is different for increasing and decreasing forces). 


for several results from experimental data]. Notice that the 
curve is non-linear, that it exhibits hysteresis (different curves 
result for increasing and decreasing forces) and that, just 
before tissue failure or tearing, the tissue stretches very little 
as the force increases. While non-linear FEM methods could 
be used for modeling this behavior, these non-linear methods 
suffer from greater computational complexity. 

Finally, standard FEM methods were developed to model 
metals and other materials with limited elasticity and plas- 
ticity. They are based on assumptions of small deformations 
with a maximum stretch on the order of 1-10% of the object 
length. In human tissue, deformations of 100-200% of the 
original tissue length are not unconmion and standard FEM 
techniques are therefore not appropriate**. 

J, 2. 2. 3-D ChainMail and elastic relaxation 
While FEM methods are proniising, current FEM formula- 
tions and software implementations are not practical for real- 
time modeling of complex geometry and detailed interior 
structure. For this reason, we have developed an algorithm 
for deforming volumetric objects based on the observation 
that complex system behavior can result when a large number 
of elements each follow simple behavior patterns (Gibson, 
1997). The tissue deformation algorithm consists of two 
processes that are applied relatively independently. Both 

**Personal communication from H. Murakami and T. Impelluso, Department 
of Applied Mechanics and Engineering Sciences, University of California, 
San Diego, CA, May, 1997. 



Figure 4. 2-D ChainMail. (a), (b) When one link of the 2-D object 
is moved along the path of the arrow, its neighboring links move 
to satisfy maximum and minimum distance requirements between 
elements. Sets of four links at the bottom of the figure show relative 
link positions when the four links are in (c) a relaxed state, (d) a 
maximally compressed stale and (e) a maximally stretched state. 


processes maintain simple relationships between elements 
and their neighbors. As in real tissue, disturbances affect only 
local elements and are propagated to other parts of the volume 
by local interactions. 

The first process of the algorithm, 3-D ChainMail, re- 
sponds to apphed displacements or forces, approximating the 
new shape of the deformed object quickly, so that collision 
detection and other physical interactions can be computed. 
Tissue elements at the site of the disturbance are moved and, 
much like the hnks in a chain, neighbors respond to a given 
node's movement if the constraints on distances between local 
nodes are violated. This concept is illustrated in the 2-D 
system of Figure 4. If a link between two nodes is stretched 
or compressed to its limit, displacements are transferred to 
neighboring links. In this way, small displacements in a 
relatively slack system result in only local deformations of 
the system, while displacements in a system that is ah-eady 
stretched, or compressed to its limit causes the whole system 
to move. Oianging the constraints on link lengths allows the 
modeling of both rigid and deformable objects. 3-D Chain- 
Mail is particularly fast for tissues with homogeneous (though 
possibly anisotropic) material properties because disturbances 
are propagated through the volume by considering each vol- 
ume element only once and by comparing each element to, at 
most, one neighbor 

The second process, elastic relaxation, adjusts element 
positions to reduce the local system energy — a function of 
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the distances between neighboring elements. When these 
distances are within some specified range, the system is in 
a low-energy state. Larger or smaller distances result in a 
higher system energy. If the forces applied to reduce the 
system energy are a linear function of displacements from 
some optimal spacing, then the system is a linear elastic 
system. By modifying these relaxation forces or allowing the 
system to have a range of optimal configurations, non-linear 
elastic and plasdc materials can also be modeled. Elastic 
relaxation is applied iteratively between applications of 3-D 
ChainMail and whenever the system is idle. Application of 
this process tends to relax the shape of the object, smoothing 
out the approximate shape produced by 3-D ChainMail. The 
result is that when an object is manipulated directly, it quickly 
deforms to an approximate shape and then relaxes to a more 
natural shape as the simulation proceeds. The time constant of 
this relaxation can be controlled by scaling forces in the elastic 
relaxation process. 

The combined behavior of the two processes exhibits a 
quasi-static deformation. While inertial and damping forces 
are not modeled directiy, the tendency of elements to not move 
unless they violate an inequality constraint gives the system 
nertial behavior, and the fact that the elastic relaxation is a 
dosed feedback loop that moves the system a fraction of the 
way towards equilibrium with each time interval gives the 
system a damped behavior. 

Both 3-D ChainMail and elastic relaxation have been im- 
plemented in 2-D and 3-D. It has been shown that deformation 
times increase at worst hnearly with the number of elements 
in the object models. In tests on a single-processor (RIOK) 
SGI Onyx workstation, volumetric objects with as many as 
125000 elements have been deformed at interactive rates. 
These tests have been performed without effort to optimize or 
parallelize code, Hiis system has not yet been used to simulate 
human tissue deformation although there are plans to adapt it 
for softrtissue in the knee model in the near future. Detemun- 
ing and validating parameters in the deformable model will be 
an important part of this next step. These issues are discussed 
farther in Subsubsection 4;2.3 in the context of knee modeling 
applications. 

33. Collision detection and response 
In addition to modeling the deformation of individual objects, 
a surgical simulation system must model interactions between 
objects, l liis requires detecting and reacting to collisio ns 
among defonn mg objects. Collision detection for voxel- 
"based objects^? straigniforward. We use a method described 
by Gibson (1995), in which the interaction space is repre - 
jnted by a. reg ular 3-Dg rid of occupancY mapceUs. In this 
approachrcach eiement ofeach object is m apped into a cell in 
the occupancy map. As an object is moved, cell contente in 



Figure 5. Simple collision detection for volumetric objects. Object 
elements are mapped into cells of the regular grid of the 3-D occu- 
pancy map. As objects are moved, cells of the occupancy map are 
updated. If an element is written into a cell that is already occupied, 
a collision is detected. 


the occupancy map are updated . A^£dlision between objects 
is detected if an elemen t i s mapped into an occupied cell in the 
occupancy map. Mgure 5 demonstrates this algoritimrin 2-D. 
Using this approach, it is possible to detect c ollisions betwe en 
both rigid and deformabl e objects and t o detect and prev ent 
self-inte rsection s. Ihis approach has^Been used interactivelv 
TodetecrcoUisions between several objects of size 64^ in real- 
time on an SGI workstation with an RIOK processor. We are 
investigating a number of methods t o imyove the speed of the 
collision-detection algorithm. These include piapping only 
gurface elements into the occupancy map, storing the object 
in a ^erarehical data structure such as octrees Joreducethe 
search space tor collision detection (He, 1997), and the use of 
a dynamic occupancy niap whose size corresponds only to the 
overlap regions of subvolumes in a hierarchical data structure 
rather than the entire interaction space. 
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In the current system, collisions are detected and object 
interpenelration is prevented but the system does not yet 
model collision response. In future systems, we intend to 
model system dynamics so that physically realistic reactions 
to collisions can be simulated. Recently, there has been signif- 
icant progress in computer graphics for real-time simulation 
of colhsion response for rigid surface models (e.g. Baraff, 
1989; Mirtich and Canny, 1995). While these methods cannot 
be applied directly to volumetric models because of the large 
number of contact points between volumetric objects, we 
believe that by averaging collision contacts over the contact 
regions rather than considering all contact points individually, 
collision responses could be calculated at reasonable rates. 
We are currentiy implementing a system for computing colli- 
sion responses between rigid volumetric models of the bones 
in the knee. 

3.4. Tissue cutting, tearing and suturing 

Modehng surgical interventions requires the simulation of 
tissue cutting, tearing and repairing or suturing. Using a 
volumetric model with links between neighboring elements, 
cutting and tearing are performed by breaking connections 
between neighbors. For cutting, these connections are broken 
along the path of the knife instrument as it is passed through 
the virtual object. Intersections between the knife path and the 
object are detected by moving the knife volume through the 
occupancy map and checking for collisions. If the knife path 
encounters a cell occupied by an element or a line linking sur- 
face elements, the appropriate neighbor connections are bro- 
ken. Tearing occurs when the distance between two elements 
is stretched beyond an allowable limit, for example, when two 
parts of an object are pulled in opposite directions. When 
a limit violation between two elements cannot be resolved 
by moving neighboring elements, the connection between the 
elements is broken. 

For joining or suturing objects, elements along the path of 
the joining instrument that have missing neighbor connections 
are paired and joined. These edge elements are detected 
by searching occupancy map cells within the vicinity of the 
joining instrument. Figure 6 shows the user interface of a 2-D 
system that allows object translation, deformation, arbitrary 
cutting, joining (or gluing), element erasing and tacking of 
elements into place. Application of this system to simulating 
cutting through knee ligaments is described below. 

4. SIMULATION OF ARTHROSCOPIC KNEE 
SURGERY 

As part of an ongoing collaboration between MERL, 
Carnegie Mellon University, Massachusetts Institute of 
Technology, and Brigham and Women's Hospital, we have 
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Figure 6. User interface for an interactive 2-D application for 
manipulated 2-D volumetric objects. Object elements can be moved 
(resulting in object translation), cut, grasped and moved (resulting 
in object deformation), tacked into place, glued together, and erased 
interactively using the computer mouse. Pointing and clicking on the 
buttons on the right-hand side of the user interface switches between 
these modes. 

built a prototype simulation system to implement and test 
some of the ideas that have been presented in this paper 
(Figure 7). The current system consists of volumetric models 
of knee anatomy created from MRI image scans, a haptic 
interface device for interacting with the knee model and 
real-time rendering for visuaUzation of the interaction. The 
current status and future plans for this system are described 
in more detail below. 

4.1. Motivation 

For our initial prototype system, we have focused on arthro- 
scopic knee surgery, a relatively new and minimally invasive 
procedure that is often used to diagnose and treat knee in- 
juries. In arthroscopy, the joint is visualized and accessed 
through small portals. An optical endoscope equipped with 
a video camera allows visualization of the procedure through 
one of the portals, while surgical probes and other instruments 
are inserted into additional portals (see Figure 8). 

We have focused on arthroscopic knee surgery for a num- 
ber of reasons. First, there are a large number of arthroscopic 
knee surgeries each year. In the US, it is estimated that knee 
procedures represent 88-90% of the approximately 1.8 mil- 
lion arthroscopic procedures performed in 1996*^. While 

^Personal communication from Alan Praemer, Department of Research and 
Scientific Affairs, American Academy of Orthopedic Surgeons, August, 
1996. 
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Figure 7. Prototype surgical simulation system. A 3-D volumeuic 
knee model has been generated from MRI images. A force-feedback 
device allows the user to haptically explore the bony surfaces of 
the knee model while real-time polygon and/or volume rendering 
provide visual feedback to the user. 


arthroscopic procedures have been shown to reduce costs 
and increase patient recovery rates, they suffer from specific 
technical limitations, namely, limited visibility through the 
arthroscope; difficulty in orienting the camera to the surgeon's 
viewpoint and restricted motion of surgical tools. Computer 
simulation will enhance the education and training of sur- 
geons, and help them deal with these technical challenges. 

Second, the knee offers an ideal initial platform for this 
project. Important structures lie in a limited volume so that 
the size of a volumetric knee model is reasonable. Many 
of the important structures in the knee are rigid so initial 
haptic and rendering implementations can be performed on 
static data sets. In addition, the soft-tissues that are of pri- 
mary importance — cartilage layers, the menisci and cruciate 
ligaments — are small enough for reasonable testing of tissue 
deformation and cutting algorithms and for performing real- 
time rendering. As faster hardware, more efficient algorithms 
and better data representations are developed, techniques de- 
veloped for the arthroscopic knee simulator will be extended 
to other systems. 

Finally, the knee has been modeled in a number of studies 
focusing on biomechanics and bio-dynamics modeling (e.g. 
Kaufman, 1988; Fijan, 1990; Blankevoortc/a/., 1991)exper- 
.nental measurements (e.g. Blankevoort et at., 1984; Fung, 
1993) and surgical simulations (e.g. Logan et al, 1996; Bajaj 
et ai, 1997). This provides a body of literature for testing 


dynamic predictions of our model, obtaining measured tissue 
parameters and comparing our results with other surgical sim- 
ulation systems. 

4.2. Prototype system: cmrrent status 
In the prototype system, object models consist of the bony 
structures of the joint, the articular cartilage, the menisci and 
the cruciate ligaments, all of which were hand-segmented 
from MRI data. The integrated system currently allows 
probing of bony structures with real-time visual and haptic 
feedback. Visual feedback is provided by volume rendering, 
polygon rendering and sectional imaging (Samosky, 1993). 

The research platform for technology development and 
the current simulator prototype consists of an eight-processor 
SGI Challenge with MIPS RIOK processors. Infinite Reality 
graphics, four RM6 raster manager boards with 3-D texture 
mapping (for interactive volume rendering) and 512 Mbytes 
of RAM. Haptic feedback is provided by a SensAble Tech- 
nologies' PHANToM with three degrees-of-freedom force 
reflection and six degrees-of-freedom sensing of position and 
orientation. The PHANToM is currently controlled via an 
SGI Indigo2 connected to the SGI Challenge with an ethemet 
connection. 

^.2.7. Image acquisition 

For the initial prototype system, a T- 1 weighted proton density 
MRI knee image sequence was acquired. The image size was 
256 X 256 X 124 with a voxel size of 0.63 x 0.63 x 0.9 oun^. 
This image was hand-segmented** into bony structures (femur, 
tibia, fibula, patella), cartilage (femoral, tibial and patellar), 
lateral and medial menisci, and anterior and posterior cruci- 
ate ligaments. These structures are illustrated in the surface 
rendered image of Figure 9. 

We have recently acquired a high-resolution knee image 
of size 512 X 512 X 90, with voxel size 0.25 x 0.25 x 
1.4 nun^. The image acquisition time was 50 min. Both Tl- 
weighted proton density images and fat-suppressed images 
were acquired. This data set has been hand segmented into 
bony structures, articular cartilage, menisci, cruciate liga- 
ments and the quadriceps and gastrocnemius muscles and 
tendons. These higher resolution models will be used to 
replace existing models in the near future. 

'^Unlike CT images, where thresholding techniques can be used effectively 
for segmentation, there arc no general automatic segmentation tools for MRI 
images. Although semi-automatic segmentation techniques can be fine tuned 
within a limited application, we do not currently have such a system cus- 
tomized for the knee. While hand-segmentation is tedious and time consum- 
ing, it has provided the models required by other components of the surgical 
simulation system. Future plans include the development of image segmen- 
tation tools specialized for knee anatomy that would reduce the segmentation 
time and facilitate the use of patient- specific models. 
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Figure 8. Illustration of arthroscopic knee suigery. The knee is 
accessed through small portals. A video endoscope inserted through 
one portal captures a live image of the surgical field arid projects it 
onto a video monitor. The surgeon manipulates surgical instruments 
through additional portals. 


Figure 9. Surface rendered image of a segmented knee. The knee 
was segmented into bony structures (femur, tibia, fibula, patella), 
cartilage (femoral, tibial and patellar), lateral and medial menisci, 
and anterior and posterior cruciate ligaments. 



Figure 10. Two-dimensional system showing soft-tissue deformation and cutting. In (b) and (c), the posterior cruciate ligament is grasped and 
pulled. In (d), the ligament is stretched and tacked into place. In (e) the ligament is cut (f) Shows how the elastic ligament tissue pulls back 
from the cut. 
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4.2.2. Deformation, cutting and suturing 

Tissue deformation, cutting and suturing have not yet been 
integrated into the prototype surgical simulation system. As 
described above, we have implemented a test system that 
performs interactive deformation of a 3-D object containing 
125 000 elements, significantly more elements than the car- 
tilage and ligaments in our knee model. Hence we believe 
that interactive deformation of these structures will not be 
prohibitively expensive. We have also implemented the 2-D 
system shown in Figure 6 that allows simulation of cutting, 
suturing and tissue deformation on a 2-D model. Figure 10 
is a sequence of images captured during interaction with this 
system. The 2-D objects consist of rigid models for the tibia 
and femur and deformable soft-tissue models for the tibial 
and femoral cartilage and the posterior cruciate ligament. In 
Figure 10b and c, the grasping tool is used to grab and pull 
on the posterior cruciate ligament. Figure lOd shows how the 
posterior cruciate ligament is stretched and tacked in place. 
Figure lOe illustrates how the ligament is cut with the knife 
tool. In Figure lOf, note that the elastic nature of the ligament 
causes its shape to relax away from the cut edge after cutting. 

Validation of the tissue models wiU be a crucial focus of the 
next phase of this project. While 3-D ChainMail has shown 
promising speed and has been shown to capture complex 
tissue behaviors, it has not yet been validated against mea- 
surements from real tissues. In prototypes of this deformation 
technique, rather than modeling particular tissues, the focus 
was on testing for specific behaviors such as rigidity versus 
deformability and elasticity versus plasticity. Parameters for 
material properties must be set and tailored to patient-specific 
data. These parameters can be derived from results reported 
in the literature or from laboratory measurements, or they 
can be hand-tuned by surgeons using the interactive system. 
Methods for validating the resultant model include compar- 
ing the model predictions to experimental data, comparing 
the resultant simulation to predictions from FEM calcula- 
tions, and user testing with surgeons and other experienced 
users. 

4.2.3. Haptic interaction 

The surgical simulation prototype uses SensAble Technolo- 
gies' PHANToM to provide force reflectance and feedback 
for haptic interaction (Figure 7). The PHANToM provides six 
degrees of sensing (3-D position and orientation) and three- 
degrees of force feedback (position only). Currently the force- 
feedback device allows the user to explore rigid object models 
using their sense of touch. We plan to provide force feedback 
during simulation of tissue deformation and cutting in the 
^e^tfjJcatQt ^ systein . The PHANToM device provides gross 
motion andforce feedback similar to what one would feel by 
grasping a pen and running its tip over a 3-D surface. Adding 


tactile feedback, using sensors such as those of Howe et al. 

(1995) , could greatiy enhance the system. 

The object models in the surgical simulation system are 
derived from segmented MRl images. Because the haptic 
resolution of the PHANToM is better than the resolution of the 
data, the binary-segmented object models must b e smoothe d 
before surface normals are calculated . If the models are not 
smoothed, the binary data causes the direction of the surface 
normals to vary widely with small changes in position, result- 
ing in unstable force feedback. 

In our prototype implementation, we use a haptic model 
similar to the density field method of Avila and Sobierajski 

(1996) . A discrete 3-D density field image was generated by 
smoothing the binary-segmented data of bony structures of 
the knee with a 5 x 5 x 5 Gaussian filter. This provided 
a smooth gray-scale image whose intensities vary from zero 
outside objects to 255 within an object. The magnitude of tiie 
feedback force is taken to be the product of a stiffness constant 
k, and the value of the density field at the PHANToM tip. For 
relatively small penetrations, the density values are a function 
of the distance from the surface of the binary volume. The 
stiffness constant was interactively hand-tuned by a surgeon 
until the interaction with the bone models felt realistic. The 
direction of the feedback force is taken to be the normalized 
gradient of the density field image, which points outward in 
the direction of the surface normal. The gradient is calculated 
using a central-difference method from trilinearly interpolated 
densities of six positions located one unit distance above, 
below, behind, in front, to the right and to the left of the tip 
position. The resultant force is 

fix, y, z) = ^ . d{x, y. z) . g{x, y, z)[\\g{x, y, z)\\T\ 

where g{x, y, z) = (d{x - 1 , y, z) - J(x + 1, y. z), d{x, y ~ 
l.z) - d{x,y -h \,z),d{x,y,z - 1) - d{x,y,z + 1)), 
and ti(x, y, z) is the linearly interpolated density at (ac, y, z), 
the PHANToM tip position. This method of calculating the 
refle cted forces requires static data and is only suitable f or 
rigid-body interactions. It does not mo del the forces jfaat 
would be reflected by elastic tissues, an issue that will he 
addressed in^ it^ifft syRt^m? 

In the current prototype, the PHANToM is controlled by 
an SGI lndigo2 while graphics and modeling are performed 
on an SGI Challenge. In order to enable interactions at rates 
greater tiian 1 kHz we store two models of the knee; a haptic 
model on the Indigo2 and a visual model on the Challenge. 
An ethemet coimection is used to exchange information about 
the PHANToM tip and orientation. This use of separate 
haptic and visual object representations is not practical for 
deformable model manipulation since changes to the models 
must be communicated over the (relatively slow) ethemet 
connection. We are currently investigating ways to control 
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a) 



b) 


Figure 11. Rendered images of knee bones in the surgical simula- 
tion prototype system: (a) volume rendered using SGI texture map 
hardware and (b) polygon rendered image of surface models. 


the PHANToM directly from the SGI Challenge to reduce this 
problem. 

4,2,4, Real-time rendering 

Interactive visualization of the simulation is essential for the 
prototype system. The current system uses twoj pproache^s 
for visualizing object models: volume rendering using th e 
3-D texture map memo ry available on our research platform 
luaprai et^L, 1995) andji gdware-accelerated polygon re n- 
deri ng of surface mode ls that were generated from the seg- 
mented images using marching cubes and triangle decimation 
(Lx)rensen and Cline, 1987; Schroeder era/., 1992). A cursor 
indicating the position of the tip of the force-feedback device 
is rendered in both approaches. Figure 1 1 shows two sample 
renderings. 

Unlike the texture map volume rendering implementation, 
polygon rendering allows us to improve the visual appear- 
ance of the rendered image by adding shading and texture. 
However, because polygonal models are not easily updated 
for objects that are deforming and changing topologically, and 
because we will require a consistent representation for physi- 
cal modeling, haptic feedback and visual rendering, we intend 
to use higher-quality volume rendering in future systems. 
We are currently investigating both software and hardware 
approaches for real-time volimie rendering. 

As part of a parallel project, MERL and Mitsubishi Electric 
are developing special purpose volume rendering hardware 
for interactive PC-based applications (Osborne et ai, 1997). 
The inmiediate goal of this project is hardware acceleration 
for volume rendering. A longer term goal is the integration 
of volumetric and siuface-based rendering in 3-D graphics 


hardware. The availability of fast rendering hardware for 
real-time, high-quality rendering of volumetric models will 
complement our activities in surgical simulation. 

While d ^ormable models have no t yet beenjuegiated 
into the prototype system, we have investigatedanumber of 
methods for rendering irregular volumes. We have imple- 
mented several of these, including raycasting (Ramamoorthy 
and ^A^helms, 1992), cell projection (Shirley and T\ichman, 
1990; WiUiams, 1992) and splatting (Westover, 1990). While 
it may be possible to integrate some of these techniques 
into hardware, they are currently too slow for real-time ob- 
ject deformation. Deformed volimietric models are currently 
visualized using OpenGL to render stirface elements as points. 
The resultant images provide enough visual feedback to guide 
the interaction, but they lack visual quality. We are currently 
investigating splatting and polygon-based rendering meth- 
ods that exploit the known connections between neighbor- 
ing surface elements to provide higher-quality rendering of 
deformable models. 

5. DISCUSSION AND CONCLUSIONS 

Volumetric methods provide powerful tools for modeling ob- 
jects with complex interior structure and tissue properties. 
In this paper, we have discussed some of the advantages 
and technical challenges that are faced in using volumetric 
methods in surgical simulation. We have also introduced tech- 
niques for modeling and interacting with volumetric objects 
and we have described an initial prototype simulator that uses 
volumetric methods for modeling arthroscopic knee surgery. 

While we advocate the use of volume methods, we readily 
acknowledge that there are many times when surface models 
are more suitable. For example, if a CAD model of a surgical 
instrument exists, there may be no advantage to converting the 
model to a (possibly) less accurate volumetric model. Hybrid 
methods that combine both polygonal and volumetric models 
for rendering, haptics and physically realistic modeling would 
be ideal. However, there are many challenges in modeling and 
rendering that must be overcome before these hybrid methods 
become feasible. In the meantime, volumetric methods pro- 
vide a powerful complement to surface graphics for modeling, 
visualizirlg and interacting with computer models. 
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